
            <!DOCTYPE html>
            <html lang="en">
            <head>
                <meta charset="UTF-8">
                <title>Prometheus Operator实战—— Prometheus、Alertmanager、Grafana 监控Springboot服务 Tomcat</title>
            </head>
            <body>
            <a href="https://andyoung.blog.csdn.net">原作者博客</a>
            <div id="content_views" class="markdown_views prism-atom-one-light">
                    <svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
                        <path stroke-linecap="round" d="M5,0 0,2.5 5,5z" id="raphael-marker-block" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0);"></path>
                    </svg>
                    <h3><a id="1_Spring_Boot__Micrometer_0"></a>1. Spring Boot 工程集成 Micrometer</h3> 
<h4><a id="11_2"></a>1.1引入依赖</h4> 
<pre><code>&lt;dependency&gt;
  &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
  &lt;artifactId&gt;spring-boot-starter-actuator&lt;/artifactId&gt;
&lt;/dependency&gt;
&lt;dependency&gt;
  &lt;groupId&gt;io.micrometer&lt;/groupId&gt;
  &lt;artifactId&gt;micrometer-registry-prometheus&lt;/artifactId&gt;
&lt;/dependency&gt;
</code></pre> 
<h4><a id="12_15"></a>1.2配置</h4> 
<p>方式一：修改配置文件<code>application.yml</code></p> 
<pre><code>management:
  endpoints:
    web:
      exposure:
        include:
          - prometheus
          - health
  metrics:
    tags:
      application: ${spring.application.name}
</code></pre> 
<p>这里 <code>management.endpoints.web.exposure.include</code> 配置为开启 Actuator 服务，因为Spring Boot Actuator 会自动配置一个 URL 为 <code>/actuator/Prometheus</code> 的 HTTP 服务来供 Prometheus 抓取数据，不过默认该服务是关闭的，该配置将打开对应或者所有的 Actuator 服务。</p> 
<p><code>management.metrics.tags.application</code> 配置会将该工程应用名称添加到计量器注册表的 tag 中去，方便后边 Prometheus 根据应用名称来区分不同的服务。</p> 
<p>方式二：添加Spring 配置 向Prometheus 添加application 的tag</p> 
<blockquote> 
 <p>management.endpoints.web.exposure.include 还是要有的</p> 
</blockquote> 
<pre><code>public class PrometheusConfiguration {

    @Value("${spring.application.name}")
    private String applicationName;

    @Bean
    MeterRegistryCustomizer&lt;MeterRegistry&gt; appMetricsCommonTags() {
        return registry -&gt; registry.config().commonTags("application", applicationName);
    }
}
</code></pre> 
<h4><a id="13__53"></a>1.3 运行查看监控效果</h4> 
<p>启动项目，访问 <strong>http://8080/actuator</strong>, 就可以看到Actuator提供的所有监控</p> 
<pre><code>{
  "_links": {
    "self": {
      "href": "http://localhost:8080/actuator",
      "templated": false
    },
    "health": {
      "href": "http://localhost:8080/actuator/health",
      "templated": false
    },
    "health-path": {
      "href": "http://localhost:8080/actuator/health/{*path}",
      "templated": true
    },
    "prometheus": {
      "href": "http://localhost:8080/actuator/prometheus",
      "templated": false
    }
  }
}
</code></pre> 
<p>参考监控数据 http://localhost:8080/actuator/prometheus</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/d13fa66a1f25a39b2caa1c39a06b4ff4.png#pic_center" alt="在这里插入图片描述"></p> 
<h4><a id="14_90"></a>1.4创建自定义监控</h4> 
<p>我们可以利用Prometheus client自定义metric</p> 
<blockquote> 
 <p>这一步非必要，只是创建业务需要的监控数据</p> 
</blockquote> 
<pre><code>@Service
public class PrometheusMeter {

    @Autowired
    private CollectorRegistry collectorRegistry;

    // 定义name为prometheus_counter的counter
    public Counter prometheusCounter(){
        return Counter.build().name("prometheus_counter").help("prometheus counter test")
                .register(collectorRegistry);
    }

    @PostConstruct
    public void init(){
        Counter counter = prometheusCounter();
        new Thread(()-&gt; {
            while (true){
                counter.inc();
                try {
                    Thread.sleep(5000);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
        }).start();
    }
}
</code></pre> 
<h4><a id="15_k8s_126"></a>1.5 运行在k8s环境下</h4> 
<p>示例工程：<a href="https://github.com/admin4j/alertmanager-dingtalk-webhook">https://github.com/admin4j/alertmanager-dingtalk-webhook</a></p> 
<p>docker: <code>docker pull admin4j/alertmanager-dingtalk-webhook</code></p> 
<p>k8s 运行yml文件</p> 
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: alertmanager-dingtalk-webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alertmanager-dingtalk-webhook
  template:
    metadata:
      labels:
        app: alertmanager-dingtalk-webhook
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - alertmanager-dingtalk-webhook
      containers:
        - name: alertmanager-dingtalk-webhook
          image: admin4j/alertmanager-dingtalk-webhook:latest
          imagePullPolicy: Always
          resources:
            limits:
              memory: 1G
            requests:
              memory: 1G
          livenessProbe:
            httpGet:
              path: /actuator/health
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 60
            timeoutSeconds: 10
            periodSeconds: 120
          readinessProbe:
            httpGet:
              path: /actuator/health
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 10
            timeoutSeconds: 10
            periodSeconds: 120
          env:
            - name: dingTalk_assessToken
              valueFrom:
                configMapKeyRef:
                  name: alertmanager-webhook-cm
                  key: dingTalk_assessToken
            - name: dingTalk_secret
              valueFrom:
                configMapKeyRef:
                  name: alertmanager-webhook-cm
                  key: dingTalk_secret
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: alertmanager-webhook-cm
data:
  dingTalk_assessToken: d54bd0d41decb444d8e716fa661c03233d10f75559a0041c60148a1de8bb802x
  dingTalk_secret: SEC7f5baedc3bc637942041077a6e5aeb4275cf403ad3c057e75b0ed8921dc2662x

----
kind: Service
apiVersion: v1
metadata:
  name: alertmanager-webhook-service
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app: alertmanager-dingtalk-webhook
  type: ClusterIP
  sessionAffinity: None
</code></pre> 
<h3><a id="2_PrometheusAlertmanagerGrafana_224"></a>2.集成 Prometheus、Alertmanager、Grafana</h3> 
<p>安装方式参考 <a href="https://blog.csdn.net/agonie201218/article/details/127965449?spm=1001.2014.3001.5501">Prometheus Operator 极简配置方式在k8s一条龙安装Prometheus 监控</a>这里就不细说了。</p> 
<h3><a id="3_Prometheus_springboot__228"></a>3. Prometheus自动发现 springboot 应用</h3> 
<p>方式一：使用Prometheus Operator 的自定义资源 CRD</p> 
<h4><a id="31__PodMonitors_232"></a>3.1 使用 PodMonitors</h4> 
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: alertmanager-webhook
  labels:
    app: alertmanager-webhook
spec:
  selector:
    matchLabels:
      app: alertmanager-webhook
  namespaceSelector: {}
  podMetricsEndpoints:
  - port: web
    interval: 15s
    path: /metrics
</code></pre> 
<p>类似地，修改 Prometheus CRD对像的用<code>spec.podMonitorSelector</code>字段。</p> 
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  serviceAccountName: prometheus
  podMonitorSelector:
    matchLabels:
      team: alertmanager-webhook
</code></pre> 
<h4><a id="32__ServiceMonitors_266"></a>3.2 使用 ServiceMonitors</h4> 
<p>与 PodMonitors 具体参考 <a href="https://blog.csdn.net/agonie201218/article/details/127965449?spm=1001.2014.3001.5501">Prometheus Operator 极简配置方式在k8s一条龙安装Prometheus 监控</a></p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/5d2f1244424efbb56c56c5bed2a97b14.png#pic_center" alt=""></p> 
<h4><a id="33__272"></a>3.3 使用注解</h4> 
<p>参考 我的 <a href="https://blog.csdn.net/agonie201218/article/details/128285943">【k8s 实战】Prometheus Operator 高级配置- 监控Kubernetes自动发现</a> 一文</p> 
<p>使用注解让 Prometheus 自动发现pod</p> 
<p>修改yml,添加注解 <code>prometheus.io/xx</code>注解</p> 
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: alertmanager-dingtalk-webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alertmanager-dingtalk-webhook
  template:
    metadata:
      labels:
        app: alertmanager-dingtalk-webhook
    annotations:
      prometheus.io/path: /actuator/prometheus
      prometheus.io/prot: '8080'
      prometheus.io/scrape: 'true'
</code></pre> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/624ab501475bb7a3a6684701809c1e9b.png#pic_center" alt="在这里插入图片描述"></p> 
<h3><a id="4_Grafana_Dashboard__306"></a>4.使用 Grafana Dashboard 展示监控项数据</h3> 
<p>去Grafana 官网找一个 springboot 监控模板 https://grafana.com/grafana/dashboards/</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/76169275ca83389ce16aa304eee7bb9e.png#pic_center" alt="在这里插入图片描述"></p> 
<p>搜索 spring boot 或者 JVM选择一个监控模板</p> 
<p>我们这里选择 12900和4701</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/f31d7d42330d1dd2ba87fddbdd676a52.png#pic_center" alt="在这里插入图片描述"></p> 
<p>12900 效果</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/386a6feddec596610b80655b25ae779c.png#pic_center" alt="在这里插入图片描述"></p> 
<p>4701效果</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/02317c1459882e1f32eb2bf4df9b1731.png#pic_center" alt="在这里插入图片描述"></p> 
<h3><a id="5_AlertManager__328"></a>5. 利用AlertManager 发送报警消息</h3> 
<p>添加<code>PrometheusRule</code> CRD</p> 
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  creationTimestamp: null
  labels:
    prometheus: k8s
    role: alert-rules
  name: prometheus-springboot-rules
spec:
  groups:
  - name: springboot
    rules:
    - alert: JvmMemoryFillingUp
      expr: (sum by (instance)(jvm_memory_used_bytes{area="heap"}) / sum by (instance)(jvm_memory_max_bytes{area="heap"})) * 100 &gt; 80
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: JVM memory filling up (instance {<!-- -->{ $labels.instance }})
        description: "JVM memory is filling up (&gt; 80%)\n  VALUE = {<!-- -->{ $value }}\n  LABELS = {<!-- -->{ $labels }}"
</code></pre> 
<p>添加成功</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/2c7741cfa1bb995c9e0e87038afe65e5.png#pic_center" alt="在这里插入图片描述"></p> 
<h3><a id="6_tomcat_359"></a>6. 监控tomcat</h3> 
<blockquote> 
 <p>因为在上面发现Grafana的Tomcat板块没有内容，很纳闷，就去官网上找了下答案，发现需要开启一下配置<br> https://github.com/spring-projects/spring-boot/issues/24503</p> 
</blockquote> 
<p>因为SpringBoot已经内置了Tomcat监控 ，需要开启一下配置就可以</p> 
<pre><code>server.tomcat.mbeanregistry.enabled=true
</code></pre> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/ad8743fa3e8e51c2c9bc789ce8b02ca0.png" alt="在这里插入图片描述"></p> 
<h3><a id="_372"></a>参考</h3> 
<p><a href="https://blog.csdn.net/agonie201218/article/details/127965449">Prometheus Operator 极简配置方式在k8s一条龙安装Prometheus 监控</a></p>
                </div>
            </body>
            </html>
            